{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "In this notebook we will continue to explore methods of learning a 1-dimensional function.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "import numpy as np\n",
                "import scipy\n",
                "import scipy.linalg\n",
                "import matplotlib.pyplot as plt\n",
                "%matplotlib inline\n",
                "from ipywidgets import interactive\n",
                "import ipywidgets as widgets\n",
                "from ipywidgets import fixed\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Let's first define some utility methods that will be useful later. `generate_x` lets us produce a set of sample points over an interval, either evenly spaced, or drawn from a uniform distribution. The other functions are simple plotting utilities to be used later.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def generate_x(n, x_type, x_low=-1, x_high=1):\n",
                "    if x_type == 'grid':\n",
                "        x = np.linspace(x_low, x_high, n, endpoint = False).astype(np.float64)\n",
                "\n",
                "    elif x_type == 'uniform_random':\n",
                "        x = np.sort(np.random.uniform(x_low, x_high, n).astype(np.float64))\n",
                "        #Note that for making it easy for plotting we sort the randomly sampled x in ascending order\n",
                "    else:\n",
                "        raise ValueError\n",
                "\n",
                "\n",
                "    return x\n",
                "\n",
                "\n",
                "def plot_function(f):\n",
                "    pts = generate_x(100, \"grid\")\n",
                "    plt.plot(pts, f(pts))\n",
                "\n",
                "\n",
                "def plot_features(featurize):\n",
                "    samples = generate_x(100, 'grid')\n",
                "    featurized_samples = featurize(samples)\n",
                "    D = len(featurized_samples)\n",
                "    for i, feature in enumerate(featurized_samples):\n",
                "        plt.plot(samples, feature, label=\"Feature {}\".format(i))\n",
                "    plt.legend(loc=\"lower right\")\n",
                "    plt.show()\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (c)\n",
                "\n",
                "In part (c), we will look at one way of orthonormalizing features under a given sample distribution. Let $p(x)$ define the probability density of the distribution of our sample points, and let $\\phi_i(x)$ and $\\phi_j(x)$ be two 1-dimensional feature functions, taking in a scalar input and returning a scalar output.\n",
                "\n",
                "The inner product of these two features over our distribution is\n",
                "$$\n",
                "    \\int \\phi_i(x) \\phi_j(x) p(x) \\mathrm{d}x\n",
                "$$\n",
                "over the range of possible values of $x$.\n",
                "\n",
                "We wish to choose features $\\phi$ that have unit norm and are mutually orthogonal. One way to do this is to use the well-known Gram-Schmidt process.\n",
                "\n",
                "First, let's look at the features of interest: the polynomial features $1$, $x$, and $x^2$.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def polynomial_featurization(x, degree):\n",
                "    n = x.shape[0]\n",
                "    A = np.zeros((degree, n))\n",
                "    for i in range(degree):\n",
                "        A[i, :] = (x ** i).reshape((1, -1))\n",
                "    return A\n",
                "\n",
                "plot_features(lambda x: polynomial_featurization(x, 3))\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "polynomial_featurization(np.array([1,2,3,4]), 3)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "These functions will help us compute the covariance of our feature vector over a probability distribution.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "N = 10000\n",
                "\n",
                "def compute_sample_covariance(features):\n",
                "    return features.T @ features\n",
                "\n",
                "def compute_feature_covariance(featurize, samples):\n",
                "    # default distribution is uniform [-1, 1]\n",
                "    if isinstance(samples, int):\n",
                "        samples = generate_x(samples, \"uniform_random\")\n",
                "    features = featurize(samples)\n",
                "    cov = features @ np.conjugate(features.T) / len(samples) # divide by N to get the expected value of the covariance for N = 1\n",
                "    return cov\n",
                "\n",
                "covariance = compute_feature_covariance(lambda X: polynomial_featurization(X, 3), N)\n",
                "plt.imshow(covariance)\n",
                "covariance\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "You should notice that the covariance of the polynomial features is clearly not $I$, so they are not orthonormal. Use Gram-Schmidt to orthonormalize these features, and fill in the relevant lines in the below code block.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def orthonormalized_quadratic_polynomial_featurization(x):\n",
                "    n = x.shape[0]\n",
                "    \"\"\"\n",
                "    The below block should look like\n",
                "    ```\n",
                "        polynomials = [\n",
                "            lambda x: <FIRST POLYNOMIAL>,\n",
                "            lambda x: <SECOND POLYNOMIAL>,\n",
                "            lambda x: <THIRD POLYNOMIAL>,\n",
                "        ]\n",
                "    ```\n",
                "\n",
                "    For instance, without any orthonormalization, it would simply be\n",
                "    ```\n",
                "        polynomials = [\n",
                "            lambda x: 1,\n",
                "            lambda x: x,\n",
                "            lambda x: x**2,\n",
                "        ]\n",
                "    ```\n",
                "    \"\"\"\n",
                "    polynomials = [\n",
                "        ### start (c) ###\n",
                "\n",
                "        ### end (c) ###\n",
                "    ]\n",
                "    A = np.zeros((3, n))\n",
                "    for i, polynomial in enumerate(polynomials):\n",
                "        A[i, :] = (polynomial(x) * np.ones(n)).reshape((1, -1))\n",
                "    return A\n",
                "\n",
                "plot_features(orthonormalized_quadratic_polynomial_featurization)\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "covariance = compute_feature_covariance(lambda X: orthonormalized_quadratic_polynomial_featurization(X), N)\n",
                "plt.imshow(covariance)\n",
                "covariance\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "If you solved the problem correctly, you should see that the covariance matrix is essentially the $3 \\times 3$ identity matrix (since we compute the covariance numerically, there may be some slight deviation due to randomness).\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (e)\n",
                "We will now consider the case when the distribution of our test points $x$ is not uniform. The `weighted_distribution` function shows how we will draw points from the interval $[-1, 1]$. Our distribution will be piecewise uniform over $[-1, 0)$ and $(0, 1]$, but each sample point will be twice as likely to be positive, as to be negative.\n",
                "\n",
                "Look at the covariance of our orthonormalized features from the previous part with respect to the uniform distribution, and with respect to this new `weighted_distribution`. Are they still orthogonal?\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def weighted_distribution(n):\n",
                "    samples = generate_x(n, \"uniform_random\", -1, 2)\n",
                "    samples[samples > 1] -= 1\n",
                "    return samples\n",
                "\n",
                "print(\"Covariance under uniform distribution:\")\n",
                "covariance = compute_feature_covariance(lambda X: orthonormalized_quadratic_polynomial_featurization(X), N)\n",
                "print(covariance)\n",
                "plt.imshow(covariance)\n",
                "plt.show()\n",
                "\n",
                "print(\"Covariance under p(x):\")\n",
                "covariance = compute_feature_covariance(lambda X: orthonormalized_quadratic_polynomial_featurization(X), weighted_distribution(N))\n",
                "print(covariance)\n",
                "plt.imshow(covariance)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We will now use an alternative method to orthonormalize a set of features with respect to an arbitrary `weighted_distribution`, as you derived in part (d) of the problem. We have implemented everything for you, all you have to do is run the below cell and understand the outputs.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def orthonormalize(featurize, samples):\n",
                "    K = compute_feature_covariance(featurize, samples)\n",
                "    K_inv_square_root = np.linalg.inv(scipy.linalg.sqrtm(K))\n",
                "\n",
                "    def featurization(samples):\n",
                "        return K_inv_square_root @ featurize(samples)\n",
                "\n",
                "    return featurization\n",
                "\n",
                "N = 10000\n",
                "\n",
                "orthonormalized_featurization = orthonormalize(\n",
                "    lambda x: polynomial_featurization(x, 3),\n",
                "    generate_x(N, \"uniform_random\", -1, 1),\n",
                ")\n",
                "print(\"Covariance under uniform distribution:\")\n",
                "covariance = compute_feature_covariance(orthonormalized_featurization, generate_x(N, \"uniform_random\", -1, 1))\n",
                "print(covariance)\n",
                "plt.imshow(covariance)\n",
                "plt.show()\n",
                "\n",
                "print(\"Orthonormalized quadratic features under uniform\")\n",
                "plot_features(orthonormalized_featurization)\n",
                "\n",
                "orthonormalized_featurization = orthonormalize(\n",
                "    lambda x: polynomial_featurization(x, 3),\n",
                "    weighted_distribution(N),\n",
                ")\n",
                "print(\"Covariance under weighted distribution:\")\n",
                "covariance = compute_feature_covariance(orthonormalized_featurization, weighted_distribution(N))\n",
                "print(covariance)\n",
                "plt.imshow(covariance)\n",
                "plt.show()\n",
                "\n",
                "print(\"Orthonormalized quadratic features under weighted distribution\")\n",
                "plot_features(orthonormalized_featurization)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Consider the above orthonormalized features under the uniform distribution, and compare them to the orthonormalization you obtained earlier using Gram-Schmidt. Think about whether they are the same, and why that is or is not the case.\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (g)\n",
                "From this point onwards, we will look at the Fourier features you studied in HW 1, rather than polynomial features. The `fourier_featurization` function featurizes scalar data points in a manner analogous to the `polynomial_featurization` function from earlier.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def fourier_featurization(x, d):\n",
                "    assert (d-1) % 2 == 0, \"d must be odd\"\n",
                "    max_r = int((d-1)/2)\n",
                "    n = len(x)\n",
                "    A = np.zeros((d, n))\n",
                "    A[0:, :] = 1\n",
                "    for d_ in range(1,max_r+1):\n",
                "        A[2*(d_-1)+1, :] =  np.sin(d_*x*np.pi)\n",
                "        A[2*(d_-1)+2, :] =  np.cos(d_*x*np.pi)\n",
                "\n",
                "    A[0, :] *= (1/np.sqrt(2))\n",
                "    A *= np.sqrt(2)\n",
                "    return A\n",
                "\n",
                "plot_features(lambda x: fourier_featurization(x, 3))\n",
                "\n",
                "fourier_featurization(np.array([1,2,3, 4]), 3)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "As we showed in HW 1, properly normalized Fourier features are orthonormal with respect to a uniform distribution over $[-1, 1]$.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "covariance = compute_feature_covariance(lambda X: fourier_featurization(X, 3), 10000)\n",
                "plt.imshow(covariance)\n",
                "print(covariance)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The calculations in the last parts of this question assume that each of our features have zero-mean. All the Fourier features except for the first feature $1$ have this property, so we will remove it from our featurization.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def zero_mean_fourier_featurization(x, d):\n",
                "    assert d % 2 == 0\n",
                "    return fourier_featurization(x, d + 1)[1:, :]\n",
                "\n",
                "plot_features(lambda x: zero_mean_fourier_featurization(x, 2))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Our goal is to use these featurizations to learn a target function, using least squares on the featurized observations. For part (g), we will assume that our target function is simply $f(x) = \\phi_1(x)$. `generate_weights` is a utility function that lets us generate random weights for use later on. `function_from_weights` converts a weight vector into a continuous function.\n",
                "\n",
                "Observe our target function and the sample points we pick for our input data.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def generate_weights(n):\n",
                "    return np.random.uniform(-1, 1, n).astype(np.float64)\n",
                "\n",
                "def function_from_weights(weights):\n",
                "    n = len(weights)\n",
                "\n",
                "    def f(x):\n",
                "        features = zero_mean_fourier_featurization(x, n)\n",
                "        return features.T @ weights\n",
                "    return f\n",
                "\n",
                "D = 10\n",
                "num_samples = 10\n",
                "true_weights = [1] + [0] * (D - 1)\n",
                "true_function = function_from_weights(true_weights)\n",
                "\n",
                "plot_function(true_function)\n",
                "samples = generate_x(num_samples, \"grid\", -1, 1)\n",
                "plt.scatter(samples, true_function(samples))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "To make things interesting, we will introduce some noise into our sample points, so that when we perform least squares we will not exactly recover the true weights. Let's add this noise, and then perform least squares on the featurized data to try and recover our original weights. Since you've already done this in HW 1, we will do it for you here.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def add_awgn_noise(y, awgn_std=0):\n",
                "    noise = np.random.normal(0, awgn_std, y.shape)\n",
                "    y_noisy = y + noise\n",
                "    return y_noisy\n",
                "\n",
                "noiseless_observations = true_function(samples)\n",
                "noisy_observations = add_awgn_noise(noiseless_observations, 0.1)\n",
                "\n",
                "featurized_samples = zero_mean_fourier_featurization(samples, D)\n",
                "\n",
                "print(\"Noisy observations vs the true function\")\n",
                "plot_function(true_function)\n",
                "plt.scatter(samples, noisy_observations)\n",
                "plt.show()\n",
                "\n",
                "# Now we will try to learn the function from the noisy observations\n",
                "estimated_weights, *_ = np.linalg.lstsq(featurized_samples.T, noisy_observations, rcond=None)\n",
                "estimated_function = function_from_weights(estimated_weights)\n",
                "\n",
                "print(\"The estimated vs true function\")\n",
                "plt.scatter(samples, noisy_observations)\n",
                "plot_function(true_function)\n",
                "plot_function(estimated_function)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Hopefully, you should see an estimate that's reasonably close to the true function. We're not really concerned with the accuracy of our estimate, however, so don't worry much if it's not so accurate. (In HW1 you studied techniques like ridge regression which can help us improve our estimate).\n",
                "\n",
                "To quantify the error of our estimate, we can compute the prediction error by computing the expected squared error in our prediction for a point drawn from the uniform distribution over $[-1, 1]$.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def sample_error(samples, true_function, predicted_function):\n",
                "    if isinstance(samples, int):\n",
                "        samples = generate_x(samples, \"uniform_random\")\n",
                "    return np.sum(np.abs(true_function(samples) - predicted_function(samples)) ** 2) / len(samples)\n",
                "\n",
                "numerical_prediction_error = sample_error(1000, true_function, estimated_function)\n",
                "print(\"Numerical prediction error:\", numerical_prediction_error)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Fundamenally, this error is caused by two things. First, by the error in our estimate of the first weight (which, in the true function, was $1$). Second, by the error in our estimate of the other weights (which were really all just $0$). We will call the first source of error \"survival\" (`SU`) and the second source \"contamination\" (`CN`).\n",
                "\n",
                "In this problem, you have shown that the error can be decomposed as\n",
                "$$\n",
                "    \\mathcal{E}_{\\text{pred}} = (1 - \\text{SU})^2 + \\text{CN}^2.\n",
                "$$\n",
                "\n",
                "Fill in the code blocks to compute the two sources of error, and verify that the above formula matches the numerical estimate of the prediction error obtained in the previous cell.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def compute_SU(true_weights, predicted_weights):\n",
                "    ### start (g)(1) ###\n",
                "\n",
                "    ### end (g)(1) ###\n",
                "\n",
                "def compute_CN(predicted_weights):\n",
                "    ### start (g)(2) ###\n",
                "\n",
                "    ### end (g)(2) ###\n",
                "\n",
                "def expected_sample_error(true_weights, predicted_weights):\n",
                "    return (1 - compute_SU(true_weights, predicted_weights))**2 + compute_CN(predicted_weights)**2\n",
                "\n",
                "print(\"SU:\", compute_SU(true_weights, estimated_weights))\n",
                "print(\"CN:\", compute_CN(estimated_weights))\n",
                "print(\"Expected prediction error:\", expected_sample_error(true_weights, estimated_weights))\n",
                "print(\"Numerically estimated prediction error:\", numerical_prediction_error)\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## Part (h)\n",
                "\n",
                "Finally, we will consider the case when our model is not powerful enough to learn the true function. Concretely, imagine that our true function is a linear combination of some $D$ features, but our prediction is a linear combination of a subset of size $d < D$ of these features. Then, it is clear that our prediction will not exactly match the true function, so there will be some error.\n",
                "\n",
                "In this section, we will look at the various components of this error, and see how they depend on $d$ and $D$. We will divide the residual $R$ between our prediction and the true value into two components: the error caused by our incorrect estimation of the weights on the first $d$ features, and the error caused by our inability to estimate the weights on the last $D - d$ features, since they are not part of our model.\n",
                "\n",
                "Concretely, in the case of zero-mean orthonormal features,\n",
                "$$\n",
                "    \\mathcal{E}_{\\text{pred}} = \\sum_{j=1}^d (w_j^* - w_j)^2 + \\text{Var}\\left(\\sum_{j=d+1}^D w_j^* \\phi_j(X)\\right) = \\sum_{j=1}^d (w_j^* - w_j)^2 + \\sum_{j=d+1}^D (w_j^*)^2.\n",
                "$$\n",
                "\n",
                "Run the below code block, and verify that our predicted value of $\\mathcal{E}_{\\text{pred}}$ matches what is calculated empirically as we vary $d$ and $\\sigma$ where $\\sigma^2$ is the variance of the AWGN in our training $y_i$.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "def compute_var_R1(true_weights, predicted_weights, featurized_samples):\n",
                "    d = len(predicted_weights)\n",
                "    return np.linalg.norm(true_weights[:d] - predicted_weights)**2\n",
                "\n",
                "def compute_var_R2(true_weights, featurized_samples):\n",
                "    d = featurized_samples.shape[0]\n",
                "    return np.linalg.norm(true_weights[d:])**2\n",
                "\n",
                "def estimate_true_function(d, sigma, true_weights):\n",
                "    num_samples = 100\n",
                "\n",
                "    D = len(true_weights)\n",
                "    true_function = function_from_weights(true_weights)\n",
                "\n",
                "    assert d < D\n",
                "\n",
                "    # uniformly sample our true function with noise\n",
                "    samples = generate_x(num_samples, \"grid\", -1, 1)\n",
                "    noiseless_observations = true_function(samples)\n",
                "    noisy_observations = add_awgn_noise(noiseless_observations, sigma)\n",
                "\n",
                "    # do least squares using the features\n",
                "    featurized_samples = zero_mean_fourier_featurization(samples, d)\n",
                "    estimated_weights, *_ = np.linalg.lstsq(featurized_samples.T, noisy_observations, rcond=None)\n",
                "\n",
                "    estimated_function = function_from_weights(estimated_weights)\n",
                "\n",
                "    numerical_error = sample_error(1000, true_function, estimated_function)\n",
                "\n",
                "    var_R1 = compute_var_R1(true_weights, estimated_weights, featurized_samples)\n",
                "    var_R2 = compute_var_R2(true_weights, featurized_samples)\n",
                "    expected_error = var_R1 + var_R2\n",
                "\n",
                "    return var_R1, var_R2, numerical_error, expected_error\n",
                "\n",
                "def plot_estimates(D, sigma):\n",
                "    # generate a true function\n",
                "    true_weights = generate_weights(D)\n",
                "\n",
                "    ds = range(2, D, 2)\n",
                "    var_R1s = []\n",
                "    var_R2s = []\n",
                "    numerical_errors = []\n",
                "    expected_errors = []\n",
                "    for d in ds:\n",
                "        var_R1, var_R2, numerical_error, expected_error = estimate_true_function(d, sigma, true_weights)\n",
                "        var_R1s.append(var_R1)\n",
                "        var_R2s.append(var_R2)\n",
                "        numerical_errors.append(numerical_error)\n",
                "        expected_errors.append(expected_error)\n",
                "\n",
                "    plt.plot(ds, var_R1s, label=\"Var(R_1)\")\n",
                "    plt.plot(ds, var_R2s, label=\"Var(R_2)\")\n",
                "    plt.plot(ds, expected_errors, label=\"E_pred\")\n",
                "    plt.xlabel(\"$d$\")\n",
                "    plt.legend()\n",
                "    plt.show()\n",
                "\n",
                "    plt.plot(ds, numerical_errors, label=\"E_numerical\")\n",
                "    plt.plot(ds, expected_errors, label=\"E_pred\")\n",
                "    plt.xlabel(\"$d$\")\n",
                "    plt.legend()\n",
                "    plt.show()\n",
                "\n",
                "slider = widgets.FloatSlider(\n",
                "    value=3,\n",
                "    min=0,\n",
                "    max=10,\n",
                "    step=0.1,\n",
                "    description='$\\sigma$',\n",
                "    continuous_update=False\n",
                ")\n",
                "\n",
                "D = 100\n",
                "print(\"D = \", D)\n",
                "interactive_plot = interactive(plot_estimates, D=fixed(D), sigma=slider)\n",
                "interactive_plot\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": []
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.6.4"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}